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ABSTRACT 

Preliminary findings of the Differential Coursework 
Patterns (DCP) Project are reported. The Project uses assessment test 
scores and transcripts from sample^ of graduating college seniors to 
determine what coursework patterns were related to gains in the 
general learned abilities of the students. Random samples of 
graduating seniors have been examined from: (l) a major private 
research university (n=105) ; (2) a state university (n=54) ; (3) a 
private college (n=146) ; and (4) a women's liberal arts college 
(n=62). Precollege Scholastic Aptitude test scores were an indication 
of the students* entering levels of learning. The Graduate Record 
Examination general tests were the post -ollege exit measure of 
general learning. Coursework pattprns resulting from a cluster 
analysis developed for the project did not produce clear distinctions 
according to academic department or major, but it was concluded that: 
(1) development of general learned abilities did not have an exact 
one-to-one relationship with departmental categories; (2) development 
of general learned abilities was not confined to the lower level of 
courses; and (3) there was little formal monitoring or description of 
the curriculum in terms of general learned abilities at an 
institution-wide level. Standardized test scores and transcripts were 
useful in the assessment of learned abilities. The relationships 
between coursework patterns and tne general curriculum suggest that 
new ways are needed to conceptualize general education in college. 
Thirteen tables, three graphs, and 10 figures present study data. A 
186-item list of references is provided. (SLD) 
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DETERMINING THE EFFECT OF DIFFERENT COURSEWORK PATTERNS 
ON GENERAL STUDENT LEARNING 
AT FOUR COLLEGES AND UNIVERSITIES 



Jr.aes L. Ratcliff 
and Associaties 
Higher Education Section 
Iowa State University 

This paper reports preliminary findings of the Differential Coursework 
Patterns (DCP) Project. These findings may have particular significance and 
interest to state policynakers^ test and assessment organizations / and those at 
colleges and universities who are responsible for academic policy as it pertains 
to undergraduate education. 

The Differential Coursework Patterns Project is a direct result of the re- 
search recommendations of the Study Group on the Conditions of Excellence in 
Higher Education. Those recommendations appeared in the Group's report^ Involve- 
ment in Learning; Realizing the Potential of American Higher Education . The pur- 
pose of the DCP Project was to use assessment test scores and transcripts from 
samples of graduating college seniors to determine what coursework patterns were 
related to gains in the general learned abilities of the students. 

We have now examined random samples of graduating seniors who began their 
undergraduate education at: 1) a major private research university^ 2) a 
doctoral-granting state university, 3) a private comprehensive college, 4) a 
women's liberal arts college, 5) a public, non-traditional liberal arts college, 
6) a comprehensive community college and subsequently transferred to a state 
university. This report excludes discussion of the public non-traditional col- 



lege due to the lack of comparability of much of the data. It also excludes dis- 
cussion of the community college data, since it comprises a sub-sample of one of 
the other institutions studied* It was not our purpose to compare student per- 
formance at these institutions. Rather ^ it was to develop a way to link what 
coursework students took to what students learned during their undergraduate 
education* 

What we have found so far, we feel, has profound implications. In this re- 
port is important information for state policymakers who may be considering im- 
plementation or revision of a state assessment program for higher education. He 
think this report can provide conservative guidance in what is and what is not a 
realistic expectation from a student assessment program within a statewide sys- 
tem of higher education. 

The preliminary findings of the DCP Project also has some important sugges- 
tions for such organizations as the Educational Testing Service, the College 
Board, the Graduate Record Examination Board, and the American College Testing 
Service. These organizations provide the major sources of standardized test in- 
formation to colleges and universities. They are currently developing new in- 
struments in response to the assessment movement in higher education. Our pre- 
liminary findings suggest some ways that these organizations can be more helpful 
through their existing instruments. 

He have been examining student transcripts and assessments of general 
learning. In doing so, we have discovered some interesting and provocative rela- 
tionships between effective coursework patterns and the overall undergraduate 
curriculum. These relationships suggest that new ways are needed for conceptual- 
izing general education in college. We propose specific ways to approach reform 
of the undergraduate curriculum. He also propose specific means for selecting 
and monitoring the effect of the curriculum on the general learning of students. 



After oaking soae bold assertions about what this working paper contains^ 
we need to also urge soae caution* In the past two years, we at the DCP Project 
have developed a Dodel of analysis and have used it to look at one saople of 
student from each of the six institutional environments listed previously. Our 
analytic sodel worked in all circumstances. It did identify the specific course- 
work associated with gains in general learning aaong college students. Yet a we 
need a second group of students froa each of these institutional environsents to 
confirh our findings. We are gathering that data now. What we present here are 
hypothesized relationships. They are provocative. But, they need further re- 
search to bear theo out. ?fhile we are exaaining a second saaple of students froa 
each of the six aforeaentioned environaents, the preliminary findings reported 
here also have taken us beyond the original scope of our investigation. In 
short, we have discovered soae aspects of student assessment and the college 
cnrriculuB which aerit further exploration. We conclude this report, therefore, 
with recoasendations for further research. In essence, we conclude by challeng- 
ing each of you — state policyaakers, testing service research analysts, col- 
lege or university curricular leaders — to join with us in a dialogue to en- 
hance our ability to understanding whac in the formal curriculum contributes to 
the general learning of undergraduate students. 



The notion of student assessment 



and some of its attendant probleas 



There is a siople notion upon which the assessment aoveaent is based: 



Assessment of learning outcomes is one solution to this lop- 
sided notion of educational quality. Instead of documenting ex- 
cellence by variables unrelated to learning^ like research 
reputation and size . . , ^ we can assess educational qual.** ty core 
directly by examining what and how ouch students actually 
learn. • • • 

Using student outcomes assessment^ institutions can refocus 
their priorities so that educational quality is based on what and 
how ouch students learn in school. (Halpern, 1987, pp. 5-6). 



Give students a test at the end of their education. If they do well on the 
test, they must have received a quality college education. If they do not, the 
college wasn't such a good investment. A simple, clear idea of quality, right? 
Not exactly! The idea is one fraught with problems. Some of the probleas have 
been discussed by others, yet bear repeating. Others come directly froa our DCP 
research. None of the problems are insurmountable, but all needed to be ad- 
dressed prior to implementing an effective assessment program. 

One problem is that students at a single college or university do not expe- 
rience the same education! Incredible as this statement seems, look at the tran- 
script evidence from four of our six institutional environments: 
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Figure 1. Description of students and their transcripts. 

SaopleiJ College 1 College 2 College 3 College 4 



NuBber of 
Students 

Percent of 
graduating seniors 



105 



lOX 



54 



15X 



146 



14X 



62 



48S 



College Calendar 



Quarters Quart.>rs Senesters Quarters 



Total number 

of .courses 

on transcripts 

Average number 
of courses taken 



5,541 
52.8 



2,087 



30.6 



6,249 
42.8 



2,607 
42.0 



Total nuober of 
Unduplicated courses 
on transcripts 

Percent of 
Duplication 

Unduplicated courses 
per 100 students 



Total nusnber of 
Unduplicated courses 
taken by 5 or more 
students 

Percent of 
Duplication by 5 
or Bore students 

Unduplicated courses 
per 100 students 



1.445 815 

73*92% 60.95X 

1,376 1,509 

303 133 

20.97X 16.32X 

289 246 



1,136 990 

81*82X 62.03% . 

778 1,597 

405 149 

35.65% 15*05% 
277 240 
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Probleas posed in examining curricular effects 
on underoraduatea 

Figure 1 shows the sanple student transcripts we examined in the DCP 
Project. The average number of courses taken by these graduating seniors sug- 
gests that froB 3,900 to 4,300 courses could be found on a set of 100 student 
transcripts! When courses which appear on laore than one transcript were counted 
only once (Unduplicated courses), the number of course.7 per 100 students ranged 
froB 778 to 1,597! Five or sore students in these samples had from 15.1 to 35.7 
percent of their coursework in cooBon. Students from the same college had very 
different formal educational experiences. 

One purpose of student assessments of general learning is to determine how 
well the college is doing in developing students' general learned abilities. So, 
if we test these students (as we did), the first assessment problem we need to 
recognize is that they did not share much of a common intellectual experience at 
the college or university they attended. That these students experienced so lit- 
tle of the curriculuB in coaicon aay be viewed as desired or undesirable. The 
fact remains, it means little to sake generalizations about the quality of edu- 
cation at the institution a s a whole when students present such a wide diversity 
of educ ational experiences on their transcripts . 



Does diversity mean disarray? 

Does this diversity of coursework mean disarray? Not necessarily. Let's use 
the quarter system as an example: It usually takes about 180 quarter credits to 
earn a B.A. degree. It is not uncommon to require 90 to 120 of these credits (33 
to 50 percent) to meet the general education requirements of the college or uni- 
versity, leaving the balance of credits to be earned in tha major, minor and 
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elect! ves. Given the variety of aajors, ainors and electives at a college or 
university, a truly randoo saople of student transcripts should produce less 
than SO percent of coursework in coamon. Thus, at least half of the formal cur- 
riculum of these graduating seniors should bd diverse. Depending upon the range 
of choices available to students in aejting their general education require- 
ments, the percent of coursework held in coanon should be equal to or less than 
one-third of coursework on the transcripts. 

The general education constitutes from one-half to one-third of the stu- 
dent's coursework. College's normally concentrate general education coursework 
in the lower division of the curriculum. They often formally or informally en- 
courage students to complete the majority of the general education requirements 
during the first two-years of college. The majority of coursework taken at a 
community college by those students intending to transfer and earn a bachelors 
is general education and elective coursework. In as much as half of the students 
earning bachelors degrees are transfer students, and because so many colleges 
and universities confine their general education requirements to the first lower 
division courses, it would seem to make sense to assess general learning of un- 
dergraduates at the end of the sophomore year — the so-called "rising junior" 
examination. 

Assessments conducted at the end of the sophomore year, however, are built 
upon the assumption that what a college intends to be the impact of its curricu- 
lum actually is the impact. If a college sets a sequence of mathematics courses 
aa a general education requirement, does this mean that student actually learn 
their advanced quantitative skills in these courses? What about the junior-level 
courses in Statistics, Accounting, Market Research, or Test and Measurement in 
Education? Don't these courses contribute as well to the general learned quanti- 
tative abilities of students? In the DCP Project research, we found this to be 
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the case; there was specific upper division coursework associated with for cer- 
tain types of general learned abilities. The difference between the general edu- 
cation coursework required by the college and the coursework patterns associated 
with gains in general learned abilities aay be attributable to the difference 
between the intended and the actual effects of the formal curriculum on general 
student learning. 

If a purpose of an assessment activity is exaaine what students learned as 
a result of their college experience, then all the curriculum - not just the 
general education curriculum ~ should be examined to determine its contribu- 
tion. If we wanted to improve the writing ability of students graduating from a 
particular college, do we enhance the freshman composition courses? Do we re- 
quire more composition courses for graduation? Do we expect writing as a re- 
quirement for all courses in the curriculum? Because different students take 
different coursework, there is no simple way to answer this question. The lower 
performance in writing may be a result of what the student selected rather than 
inherent weaknesses in courses with the intent of improving one's writing 
skills. One of the exciting results of the DCP Project research is that we can 
identify the coursework p atterns of tho se student.^ who did imornve in specific 
qeneral learned abilitie s. A major gap in the assessment movement has been the 
lack of a method for linking the coursework in which student enrolled with gains 
in specific types of general learned abilities. 

Because there has not been a means for linking specific coursework with 
specific types of learning, there As no means for arriving at the conclusion 
that a required core of courses or a distributional menu of courses is superior 
in conveying general learning to college undergraduates. Diversity in the cur- 
riculum has not been shown conclusively to have either an affirmative or detri- 
mental effect on students' cognitive development. This diversity in the under- 
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graduate curriculua does have iBplications for assessing student learning. To 
assess the effect of the foraal curriculuB on general student learning, we »ust 
first be prepared to identify what that curriculua was. And we can't just do 
that by taking about the institution aa a whole or its general education re- 
quire.ents alone. Student experiences with the curriculum are siaply too di- 
verse. Instead, our .ethod of analysis .ust allow for a natural level of diver- 
sity in the curriculum, in student interests and abilities, and in the resulting 
selection of courseworJc patterns. Such an allowance for dive.-sity adds to the 
coaplexity of the task of linking coursework patterns with gains in student 
learned abilities, but coping with such coaplexity and diversity is not insur- 
aountable. 

Testing how auch students learned as undergraduates without looking at what 
they took in college is a bit like exaaining the fuel efficiency of the college 
■otor pool fleet without determining how aany dump trucks and a how many subcon- 
pact cars are in the fleet. To deter.ine the effects of college on student ' 
learning, the unit of analysis has to be the coursework taken, not the college 
as a whole. Siailarly, to assess student learning in college without acknowledg- 
ing the student's prior learning in high school, eleaentary school and preschool 
is to aeasure the college's selectivity in adnission, rather than the gains it 
contributed to the student's cognitive developnent. 

What co nstitutes a coursework pattern? 

The prevalent way to view the college curriculum is by its intentions, 
rather than by its results. Given that measuring the effect of the curriculum 
is problematic, it is not surprising that many studies presume, rather than test 
the effect of different patterns of coursework. 
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The college curricuJu. ia substantiative, additive and ta.poral. In teras 
of cognitive theories of curriculum development, both content and process con- 
tribute to developaental learning in students (Tyler, 1950; Taba, 1962) . Essen- 
tialist and constructionist theories of curriculum stress coabinations of sub- 
jects (core curricula, great books, etc.) as influential on general learned 
abilities of college students (Fuhroann and Grasha, 1983). The medieval univer- 
sity curriculuB was organized according to combinations and sequences of courses 
as well as individual subjects (Rudolph, 1977); the seven liberal arts were se- 
quenced into the prerequisite subjects of the guadriviua (arithmetic, geometry, 
astronomy, music) and the higher order subjects, the triyium (logic, grammar, 
and rhetoric). Together, the quadriviua and trivium provided an individual with 
the general learned abilities needed to study the three philosophies of Aristo- 
tle: natural philosophy (physics), moral philosophy (ethics), end mental philos- 
ophy (metaphysics). These combinations and sequences of coursework have been 
generalized to more recently into concepts of breadth and depth as criteria by 
which to describe higher education curricula (Blackburn et al., 1976). 

While the notion that combinations of concurrent cou-sework and that devel- 
opmental sequences of coursework load to effects in the general learned abili- 
ties of students is derived from the medieval university, it is underscored and 
further supported by the research of contemporary developmental theorists. Per- 
ry (1968) for example, stated that development 'consists of an orderly progres- 
sion of cognition in which more complex forms are created by the differentiation 
and reintegration of earlier, simpler forms.' (Perry, 1968, p. 44). 

The value of curricular substance and sequence are presumed in formulations 
of core curricula, in the four levels of study (freshman, sophomore, junior and 
senior years), in the corresponding practice of assigning course numbers accord- 
ing to those divisions, and in the practice of assigning course prerequisites. 
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To assess the lopact of these coursew6rk patterns on the general learned abili- 
ties of students^ the additive^ substantiative and sequential characteristics of 
student course-taking need to be examined. These notions of what ought to be 
taught and what students ought to learn presuoably represent the philosophical 
and educational aims of the particular college. 

Nevertheless/ a distinction should be aade between those patterns of 
coursework intended to fulfilled undergraduate program and degree requirements 
and those patterns of coursework which students actually choose (Boyer & Ahl- 
gven, 1981/ 1982/ 1987; Warren/ 1986). Intentional patterns of coursework are 
provided in a variety of publications issued by tho institution: the college 
catalog/ the annual schedule of tiaes and days of courses/ and program descrip- 
tions issued by departaents and divisions within the college. Richardson et al. 
(1982) provide evidence that a minority of students aay consult these statements 
of curricular intent prior to making decisions about which courses to choose. 
Other forms of intentional coursework patterns are the lists of courses or sub- 
jects required for certification or licensure in a particular profession/ occu- 
pation or technical field. Such lists of coursework may be compiled by practi- 
tioners and academics of a given discipline or profession to accredit college or 
university programs. Just as the curriculum of a particular college may repre- 
sent the philosophy and educational aims of that institution/ so too may the 
certification/ licensure and accrediting standards articulate the intentions of 
state/ regional/ disciplinary and programmatic associations. All are intended 
patterns of coursework in the curriculum whose measure of effectiveness/ in 
part/ is the extent to which these patterns accomplish their aims in practice. 

In a college curriculum/ a single course is the smallest unit of analysis. 
In assessing the impact of the curriculum on the general cognitive development 
of students/ the course constitutes a datum in the analysis. A pattern of data 
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is a design resulting froo 'the relation aoong a set of objects" (Roiaesburg^ p." 
278). In this case, the objects are courses. Therefore^ a coursework pattern is 
a design resulting froa relationships aoong courses. A cluster of courses is a 
set of one or sore courses found to be similar to one another aceording to a 
given set of attributes. In the DCP research^ courses whose students 
demonstrated comparable GRE item-type residuals were grouped together. Thus^ 
for the DCP project^ a cluster of courses wat a set of courses associated by 
common effects on the general learned abilities of the students enrolling in 
them. Stated another vay, a cluster of courses is a pattern based on the actual 
enrollment of students^ rather than the intended enrollment pax;tern of the col- 
lege or university. This distinction is important in order to differentiate be- 
tween the consequences of the college curriculum and its intents. 

What are general learned abilities? 

There is widespread disagreement on what constitutes "general learned abil- 
ities"^ and that disagreement is manifest in the variety of general education 
goals and degree requirements found in American higher education (Bergquist^ 
Gould & Greenberg^ 1981; Carnegie Foundation for the Advancement of Teaching^ 
1979; Gaff^ 1983; Levine^ 1981). Within the term "general learned abilities"^ we 
mean to include such frequently used terms as "higher order intellectual pro- 
cesses" (Pascarella, 1985), "academic competencies" (Warren, 1978), "generic 
competencies" (Ewens, 1979), "generic cognitive capabilities" (Woditsch, 1977), 
and "^general academic ability" (Conrad, Trisman & Miller, 1977). Disagreement on 
terminology is but one aspect of. the problems associated with measuring the gen- 
eral learning of students as undergraduates. 

Current notions of how to assess college outcomes call for multiple mea- 
sures of student achievement. No one measure has been found to accurately re- 
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fleet the variety of definitions of general learning and cognitive developaent^ 
the aixture of curricular goals and institutional characteristics found across 
the landscape of higher education and among the diversity of instructional pro- 
cedures and curricular organizations of undergraduate higher education. The re- 
sult has been a call for multiple measures of assessment of student learning. 
Policymakers and academic leaders tend to believe that since colleges and uni- 
versities have broad missions and goals^ assessments should be comparably broad 
enough to provide evaluation about as many institutional intents as possible 
(Loacker^ Cromwell & CBrien^ 1986; Nettles^ 1987). 

A common stumbling block in the development of an assessment program is 
that of what form of test or assessment information to use. It is acknowledged 
that there is no clear conception of what constitutes general learning, either 
in the college curriculum or in the various tests and assessment devices. If a 
college attempts to reach consensus among its constituents on either general 
education goals or on the "best" measure of general learned abilities, there 
surely will be heated discussion and the quest may end in irresolution or, 
worse, abandonment of the assessment initiative. Rathering than searching for 
the ideal reasure of general learning in college, those charged with assessment 
can better direct their energies toward the selection of a constellation of 
a ssessment means and measures which appear to be appropriate criteria for 
describing one or more dimension of the general learning goals of the college > 
As will be seen, the DCP Project research design provides a basis for 
determining the relative extent to which each in a set of measures sxplains 
student learning within a specific college environment. 

Given the confusion of terms, intents and theoretical framsworks given to 
explain "general learned abilities", we developed an analytic model for the DCP 
Project research design that is criterion-referenced . That is, the model is not 
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baaed on any one notion of uhat constitutes "general learned abilities". It is' 
not dependent on any one college's curricular structure or its intended goals to 
proaote student cognitive development. The DCP analytic aodel allows the. use of 
multiple and different aeasures of student outcoaes. It provides a aeans for 
weighing those various aeasures to determine the extent to which they reflect 
the courseworJc in which the students actually enrolled. The validity of the as- 
sessnents (aultiple aeasures) of student learning rests on the validity of the 
outcoae aeasures selected, not on the DCP analytic aodel. 

The DCP aodel fulfills the need for multiple aeasures and criterion-refer- 
enced measures of student learning. However, the design described below is not 
dependent upon any given college curricular structure or organization and, in 
fact, was tested in six very different higher education institutions. It is in 
free of bias engendered by specific institutional goals or preconceived notions 
of the curriculum, such as breath or depth of the general education coursework 
(Cronbach, 1985, p. 212). Thus, "general learned abilities" — the tsra used to 
denote student cognitive developaent in the DCP Project — is defined by the 
criteria selected to measure general student learning. Such criteria, by neces- 
sity, should encompass a variety of commonly recognized areas of knowledge, 
skill and ability developaent in undergraduate education. 

What constitutes student acfaieveaent? 

While the expansiveness and diversity of the college curriculum has impli- 
cations for determining what were the college effects on students, there are 
also a number of problems associated with measuring "general learned abilities." 
The first of these is establishing exactly what constitutes growth or gains in 
such abilities. Simply measuring how graduating seniors perform on a series of 
tests is not a sufficient basis for generalizations about the importance of col- 
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lege. The assesscent of student outcoaes is heavily effected by the students' 
acadeaic achieveaent prior to entering college (Astin^ 1970a^ 1970bj Bowen^ 
1977; Nickens^ 1970). In fact^ standardized tests used for college admission^ 
such as the Scholastic Aptitude Test (SAT), have been shoi^n to be strongly cor- 
related with tests used for graduate and professional school adaissions, such as 
the General Tests of the Graduate Record Exaaination (GRE). These correlations 
have been demonstrated for the total and sub-scores on the two tests, suggesting 
that a large proportion of what postcollege tests, such as the GRE, aeasures are 
attributable to student learning prior to college. 

Nichols (1964) studied pre-college and post-college test scores of 381 Na- 
tional Merit Scholars, using SAT-Verbal and SAT-Hath scores as student aptitude 
aeasures; these scores correlated strongly with the GRE- Verbal and GRE- 
Quantitative (r«=.74), while student/faculty ratio, library books per student, 
average ability level of the student body, and affluence of the college were all 
unrelated to GRE scores. Nichols concluded that "the college a student attends 
does, indeed, have an effect on his perforaance on an exaaination such as the 
GRE" (p. 9). 

Rock, Centra and Linn (1970) and Rock, Baird and Linn (1972) examined SAT 
and GRE area test scores of 6,855 students who graduated from ninety-five col- 
leges, predoainantly saall, private liberal arts institutions. The correlation 
between college aeans on SAT-Verbal and GRE-Total was 0.91. However, the col- 
leges whose students had the saae SAT aeans did not necessarily have siailar GRE 
aeans. Rock, Baird and Linn found that "for colleges characterized by siailar 
and relatively higher verbal input, the huaanities data do suggest that propor- 
tion of faculty with the doctorate, size of budget, and selectivity are related 
to achieveaent* (p. 158). 
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That the standardized precollege and postcollege tests, such as the SAT and 
GRE, are strongly correlated should not be surprising. Students typically bring 
12 or Bore years of foraal education with then upon entrance to college, and 
since the college years traditionally constitute 4 or 5 years, a large propor- 
tion of general learned abilities of students should be attributable to their 
learning prior to college. So we should anticipate strong relationships between 
well-established aeasures of general learning, such as the SAT and GRE, because 
both aeasure coaparable types of learning and a large proportion of that learn- 
ing occurs prior to the students adaission to college. 

Unfortunately, we have found sooe assessment programs have not taken into 
account student learning prior to college. With a high proportion of general 
learning occurring prior to college, and with no aeans to account for prior 
learning in such assessnent prograas, colleges run the danger of confusing the 
achieveaent of the students with the selectivity of the adsissions process. If 
one college adaits higher ability students than a second, the assessment results 
at the first college will be higher than that at the second sisply by virtue of 
the relative nuaber of "grey cells' students brought to the occasion, not neces- 
sarily the learning they acquired during their college years. While Astin (1985) 
has Bade this point repeatedly in advancing his concept of "talent development" 
among college students, it is also important in the use of assessnent for pro- 
graa review at or between institutions. Higher education academic policy and 
resources should be directed by accoaplishaents, not by the extent to which par- 
ticular institutions can attract students to enroll. 

We have noted in professional aeetings and in the literature a general an- 
tipathy toward standardized tests as measures of general learned ability. The 
tests have been criticized for not measuring "critical thinking" and "higher 
order reasoning skills". Furthermore, test scores as reported have little diag- 
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nostic value; their very structure and procedures are Intended to be predictive 
rather than diagnostic. The SAT serves to predict how well a student will per- 
form in college in oath and verbal abilities. The 6RE General Test offers a 
forecast of how well a student will perform in graduate school in verbal, quan- 
titative and analytic abilities (the three sub-scores of the test), based on 
th«ir general learning through their years as undergraduates. 

In the DCP Project, we used students' precollege SAT scores as indicators 
of their entering levels of general learning. We use the General Tests of the 
Graduate Record Examination (GRE) as the postcollege exit measures of general 
learning for the samples of graduating seniors. 

The Normative Base of the Graduate Record Examination 

The strong relationship between the SAT and the GRE is both an asset and a 
liability. The use of the SAT sub-scores as pre-college measures and the GRE 
item-types as a post-college measures does provide a basis for controlling the 
effects of student academic achievement using comparable definitions of general 
learned abilities and comparable testing procedures. However, the strong corre- 
lation between the two tests leaves only a small amount of explained variance 
between pre-co?^ege and post-college scores to attribute to general learning 
associated with a baccalaureate program. 

The dilemma posed by the use of the SAT and GRE as measures of general 
learned abilities among students is exacerbated by the student population dif- 
ferences upon which the tests are normed. Adelman (1985) estimated that 25X-30X 
of graduating seniors take the GRE General examinations, while 60% of the grad- 
uating high school students take the ACTs or SATs. These rough percentages are 
understandable since fewer individuals choose to continue their education from 
bachelors to graduate study than do those who choose to go to college from high 



school. Nevertheless, a conaequence is that the 6RE exaainations are norned on 
a higher ability population than the SATs (Pascarella, 1985). The individuals 
taking the GRE exaainations constitute a self-selected sample, driven in part by 
the requirenents of graduate schools, professional schools, departments offering 
graduate degrees, and organizations requiring such examinations as part of the 
foraal application for fellowships and scholarships (Adelnan, 1985). Thus, Gra- 
duate Record Exaaination can be accurately viewed as a measure of a students' 
predicted performance in graduate school as well as a taeasure of that students' 
general acadeaic accoBplishaents as an undergraduate. Exaaination of fiv9 
institutional samples froa the DCP Project revealed the students were roughly 
coaparable to the nora of graduating seniors at each college in terms' of SAT 
scores, and distribution of aajor across the curriculua. Since the analyses were 
institution specific and criterion-referenced, the role of the noras upon which 
the GRE were based did not iaaediately enter into the analysis. 

The Graduate Rec ord Exaaination as a aeasure 

of higher order reasoning or critical thinking ability 

The GRE and SAT tests have been criticized for a) the bias resulting froa 
groups upon which they were noraed (Adelman, 1985; Nettles, Thoeny 4 Gosman, 
1986) and b) their liaitation in measuring higher order reasoning skills. It is 
argued that a aajor function of undergraduate education is the development of 
higher order reasoning or critical thinking skills and that multiple choice, 
paper and pencil tests do not measure these skills well Yet, these two concepts 
elude any concise definitions (Skinner, 1976). 

One factor differentiating tests of critical thinking is that of problem 
structure (Sternberg, 1982; Wood, 1983). Problem structure is the extent to 
which a problem can be described fully and can be answered rightly or wrongly. 
Complex social, political or economic problems do not have right or wrong an- 
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swers and often there very nature is debated. These are ill-defined problem 
sets. In contrast/ probleaa which can be solved by deductive logic (in the spir- 
it of Sherlock Holoes or Miss Marple) possess a high degree of certainty and 
correctness; they are well -structured probleas (Churchnan^ 1971). 

Two popular measures of critical thinking are the Cornell Critical Thinking 
Test (CCTT) and the Watson-Glaser Critical Thinking Appraisal (H6CTA). Each mea- 
sures student's ability to solve structured problems (Ennis & Millman, 1971; 
Watson & Glaser, 1964). Each has been shown to have strong correlations with the 
ACT/ SAT and 6RE examinations (Little^ 1973; Bennett^ 1975/1976). In a recent 
study/ for exaaplO/ King/ Wood & Mines (1988) found that the W6CTA correlated 
with the ACT at r»=.59 and the CCTT correlated with the ACT at r«=.62. If sixty 
percent of the variance in scores on a college-level critical thinking test is 
explained by a traditional/ standardized measure of student achievement/ what is 
the critical thinking test really aeasuring? The difference between what the 
standardized measures of general learning measure and what is measured in sev- 
eral of the tests of critical thinking may not be fundamentally different. 

Students in the DCP Project samples showed significant improvement in Ana- 
lytic Reasoning (ARE) and Logical Reasoning (LR) abilities. To what extent do 
these item-types represent critical thinking or high order reasoning skills? 
Further investigation is warranted to determine the nature of the knowledge and 
cognitive abilities required to answer these questions. To what extent do they, 
for example/ suffice a college's question to measure the development of critical 
thinking abilities in its students? 

The 6RE and SAT tests have been criticized for a) the bias resulting from 
groups upon which they wore normed (Adelman/ 1985; Nettles/ Thoeny & Gosman/ 
1986) and b) their limitation in measuring higher order reasoning skills. These 
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criticisms notw.lthstandlng, .the GRE and SAT tests do provide an econoalcal^ 
practicieie and valid way of measuring selected general learned abilities while 
controHing for the incc^ing acadealc accoraplishoents of freshaen (and wooen) 
undergraduates (Astin^ 1968; Wondel^ 1977). Critics of the GRE and SAT as mea- 
sures of general learned abilities attack the validity of the measures them- 
selves. These criticisms primarily are based on the use of sub-scores and total 
scores of the teste; the use of the item-type scores on either the GRE or SAI as 
multiple measures of general iGarning have not been widely explored (Adelman^ 
1988). 

Research in the DCP Project suggests that the it^im-types of these tests can 
bo used effectively to describe specific general learned abilities across a 
broad and representative spectrua of graduating eeniors. Ihaae ^salors come from 
selective and open-admissions institutions of higher education. While high abil- 
ity students who demonstrated significant improvement in one or more general 
learned abilities did enroll in different coursework patterns from students of 
low entering ability^ there was little evidence to suggest that the GRE test 
failed to differentiate between the pre- and post-college achievement of high 
ability students. There were not significant instances where the GRE item-type 
score predicted by the student's SAT score exceeded the actual perfect score for 
that iteortype. In no case did a individual student's 9 predicted GRE scores 
exceed the 9 possible GRE item- type scores. 

Development of the DCP model of analysi^i 

The GRE and SAT tests were used in the development of the DCP Project as 
pre-college and post-college achievement measures. However / the analytic model 
developed is not dependent upon these sets of measures. The model could be em- 
ployed using any set of correlated pre-tests and post-tests. What is needed for 
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the aodel to function effectively is multiple measures of student learning 
wherein pre-college student achieveoent is accounted for and controlled. For 
pragaatic and economic reasons, the cluster analytic aodel tras developed using 
SAT and GRE tests, respectively^ as pre-test and post-test aeasures of the gen- 
eral learned abilities of baccalaureate candidsUtes for graduation. The aodel was 
initially developed using the two sub-tests of. the SAT and GRE: thei SAT Verbal 
Teist (SAT-V), the SAT Mathematical Test (SAT-H)^ the GRE Verbal (GRE-V), and the 
GRE Quantitative (6RE-Q). Subsequent development and testing of the aodel ea- 
ploy€;d the 9 itea-type parts of the GRE as multiple measures of student learn- 
ing. Therefore^ the cluster analytic model will be described in terms of 9 mea- 
r.ures of student general learned ability. There was prior research to suggost 
that the 9 i tea-type scores were independent me^^sures of general learning. 

Sllson (1985) examined the criterion-validity of the 9 itea-type part 
scores of the GRE General Test to the prediction of self-reported undergraduate 
grade point average. For hih research, Wilson used the GRE scores of 9,375 exaa- 
inees in 9 different fields of «tudy representing 437 undergraduate departments 
froB 149 colleges and universities. D^kta were £i;;st standardized within each 
department, then pooled for analysis by field of study. Rei^ults suggested that 
the GP'::^' aloa-type scores did differentiate undergraduate GPA by field of study. 
This and other studies (Powers, Swinton & Carlson, 1977; Swinton & Powers^ 1982; 
Wilson, 1974) indicate that the 9 item-type subparts of the GRE test measure 
different and somewhat unique general learned abilities. 

The GRE General Test consists three sections j verbal, quantitative and ana- 
lytical; within each test section are specific types of test iteas (i.e. Verbal: 
analogy item; Ouantitative: quantitative comparisons; Analytic: logical reason- 
ing). There are 9 itea-types within the General Test; the residual differences 
between the observed GRE scores (post-college measure) and the GRE scores pre- 
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dieted by the students' corresponding SAT scores (pre-college measure) were used 
to gauge general learned abilities attributable to the students' undergraduate 
education. For the purposes of development and testing of the cluster analytic 
aodel, the residual differences between the observed and predicted values of 
each GRE item-type served as the 9 measures of student gains in general learned 
abilities during the time in which the student was enrolled in college. Hereaf- 
ter, these 9 item-type residuals Hill be referred to as student qore_3ains. 
Thus in an economical and practical way, these student score gains represent a 
set of multiple measures of general learned abilities which account for and con- 
trol the effect of pre~colle«e student achievement on post-college student out- 
comes. 

The advantages of using residual scores rather than mean scores in the de- 
sign of assessments of undergraduate learning is illustrated in the following 
comparison of the performance of seniors at two colleges in the DCP project: 



Fig 2. Residuals vs Means 
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Dlacuasion of aerlta of score residuals over siMple score aeana 
in deterainlng gains In general learned abilitieg 

Comparison of the eean GRE itea-type scores and the GRE residual scores 
(once the effect of the SAT is reooved) illustrate the differences between con- 
trolling and not controlling for incoaing student ability. Consider the follow- 
ing data froa the DCP research: College 1 has extreoely selective aduissions. 
Both the verbal and quantitative SAT scores are quite high at College !• College 
3/ while aaintaining selective admissions as well/ has st.jdents entering with 
considerably lower SAT quantitative scores • Figure 6 presents the means (lines) 
and residuals (bars) for each college's seniors. 

If we examine simple mean score performance on the GRE (and therefore/ do 
not take into account the precollege abilities of the students)/ we conclude 
that College 1 seniors evidenced greater skill in most all areas. Their skill in 
making Quantitative Comparisons/ Analytic Reasoning (ARE) and Reading 
Comprehension (RD) were particularly pronounced. One might falsely conclude from 
examination of simple aean scores that students in College 1 learn more than 
those from College 3. 

When we exaoine residual scores (after the effect of SAT scores are re- 
moved)/ we find quite a different story • College 1 students showed significant 
iaproveaent in verbal abilities/ particularly Analogies (ANA) and Reading Com- 
prehension (RD). College 3 students showed significant gains in quantitative 
abilities/ particularly in Data Interpretation skills (DI). Students at both 
colleges showed signific-^nt improvement in all measures of general learned abil- 
ity/ with the residual scores representing froa 45 to 92 percent of their learn- 
ing as measured by the GRE itea-types» Residuals tell you more about the perfor- 
mance of students/ and they account for their learning prior to college. 
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Figure 2a conpares the aean score residuals for each college saople and 
each GRE iteis-type. All colleges deaonstrated iaprove.ent in the general 
learning of their students. College 4 showed particularly large gains in Verbal 
lte>-types.- College 3 exhibited gains in Quantitative abilities, particularly in 
Data Interpretation. College 2 aanifested .arkad perforaance iaprovement in 
Logical Reasoning. Each college demonstrated a different mix of score gains 
anong the 9 criterion variables. The residual scores provide a basis for 
deteraining the extent to which each measure of general learning explains the 
achievenents of a group of college students. 

To reconfir. the findings of Wilson (1985), we coapcred the GRE iten-type 
residuals and GRE sub-score residuals. If the itea-types meas^e discrete type, 
of learning, then a greater proportion of the sub-scores should be explained by 
the SAT than the itea-type scores. In short, the residuals for the GRE 
itea-types should be greater that those for the ORE sub-scores, indicating that 
less of the variance in itea-type scores is explained by the SAT. As the 
coaparison of three college saaples in the DCP Project shows (see Figure 3), 
itea-types do evince greater residuals ^^an do sub-scores. 
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Fig 2a. Cowpaipison of residuals 
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Figure 3 shows that College 4 students showed greater gains from precollege to 
postcollege measures in verbal abilities. Improvement in student ability to an- 
swer Antonym (ANT) and Reading Comprehension <RD) questions were particularly 
pronouncad. College 3 students demonstrated large gains in quantitative skills. 
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Both College 3 and College 4 showed considerable inprovement in Data Interpreta- 
tion (DI), while College 2 students shewed exceptional gains in Logical Reason- 
ing (LR) skills. All four colleges evinced larger gains in specific abilities in 
individual GRE iten-types than in the aggregate GRE sub-scores (GRE-V, GRE-Q & 
GRE-A) . 

Two observations are iaportant to note here. First, the item-tvpes scores 
appeared to aeasure discrete fora s of learning . Secondly, while all colleges 
showed significant iaproveaent in all areas, itea-tvpe residual gains did vary 
by college . College environments were associated with gains in the general 
learning of students. Different college environments were linked with different 
patterns of improvement. And the GRE item-types did differentiate this learning 
more effectively than did GRE sub-scores. 

Given this confirmation of Wilson's findings regarding item-types, it would 
seem warranted to investigation SAT item-types as well. Wilson asserted that 
they too were associated with specific types of learning. If such were the case, 
a more effective set of multiple measures of precollege learning could be used. 
Unfortunately, these SAT item-type scores were not available to the DCP Project 
teaa. 

Why should item-types have greater power in describing discrete sets of 
learning than aggregate sub-scores? This is partly the result of the way the 
tests were constructed. If the GRE included but one type of mathematics ques- 
tions, it would be subject to the criticism that it was not a. comprehensive mea- 
sure of students' quantitative skills. So several discrete types of mathematics 
questions were included in the GRE. Using factor analysis, the creators of the 
test established the construct validity of each item-type. When item-type scores 
are aggregated to form a single Quantitative sub-score to predict academic per- 
formance, the discrete qualities of each item-type are submerged in the large 
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Attributing student score gains to courses 
Reliability and correlation of 6RE Item-types 

Before reaoving the effects of the students' SAT scores froa their GRE 
itea-type scores^ we first tested the reliability of the GRE iteia-types for each 
saaple of college seniors. We wanted xo deteraine if GRE itea-types were 
reliable ceasures of student learning for each group of students. Hext^ we 
exaained the correlation between the GRE itea-types and the SAT sub-scores and 
total scores. We did this to sake sure we were using the aost appropriate SAT 
score or sub-score as a control variable. Finally ^ we conducted a regression GRE 
itea-types on SAT sub-scores to calculate student gain scores (residuals) for 
each GRE itea-type. He wanted to see if Wilson's (1985) assertion the 
itea-types measured discrete types of learning (and hence constituted aultiple 
aeasures of learning) was affiraed in the sample data froa the various sample 
groups. If such were the case^ there should be greater unexplained variance 
(residuals) in the GRE itea-types than in the GRE sub-scores. 

Our first question was whether GRE itea-types evinced reliability within 
each saapl^ group. The results of the reliability analysis for each college 
saaple is presented in Figure 4. The reliability of the individual itea-types 
tended to increase with the number of items comprising the given item-type. 
While those itea-types comprised of a -small number of items (Data Interpretation 
and Logical Reasoning) presented low alpha coefficients^ overall the GRE 
itea-types demonstrated satisfactory levels of reliability^ stability and 
accuracy of measurement. ~- 
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coefficients of GRE Ite^- types for 3 colleges. 



GRE Ite«-types 



Cronbach's Alpha 



Code 


Number 
of iteas 


College 1 
n = 107 


College 3 
n = 62 


College 4 
n = 146 


ANA 


18 


.5674 


fil 9fi 


.4414 


SC 


14 


.6192 


.5980 


.5317 


RO 


22 


.7617 


.7013 


.5826 


AHT 


22 


6769 


Q OCT 


.6054 


1 QC 


30 


.7814 


.6968 


.7475 


RH 


20 


6650 




.7023 


DI 


10 


5786 


• loX/ 


.3565 


ARE 


38 


.8055 


.8021 


.7931 


LS 


12 


.5034 


.6065 


.3602 


GRE-V 


76 


.8765 


.8852 


.7943 


GRE-Q 


60 


.8686 


.7575 


.8405 


GRE-A 


50 


.8315 


.8261 


.8091 



Analogy 

Sentence Coapletlon 
Reading Comprehension 
Antonyas 



Regular Matheaatics 
Data interpretation 

Analytical Reasoning 
Logical Reasoning 

GRE Verbal 

GRE Quantitative 

GRE Analytic 



In the DCP Project cluster analytic nodel, the SAT sub-scores are used as 
aeasures of entering student ability. Prior, to regressing GRE itea-type scores 
on SAT scores, it is iaportant to deteraine the extent to which GRE item-types 
and SAT sub-scores are correlated. Deteraining whether the GRE itea-type. 
Analogies, for example, has a stronger correlation with SAT Verbal, SAT Math or 
the total SAT scores will help deteraine which SAT score should be used in the 
subsequent regression analysis. 

As Figure 5.,. generally the strongest relationships were between the GRE 
Verbal itea- types and the SAT Verbal score. In the case of College «l, GRE 
Reading Comprehension was more strongly correlated with the SAT Total score than 
with the SAT Verbal sub-score. Highest correlations occurred between the GRE 
Quantitative Item-Types and the SAT Math score, and between the GRE Analytic 
itea-types and the SAT Total score. In the instances of Colleges #3 and #4, Data 
Interpretation was as strongly correlated with the SAT Total Score as with the 
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Math sub-score. And in the cases of College #3, Analytic Reasoning was about as 
strongly correlated with the SAT Hath sub-score as it was with the SAT Total 
Score. 

The GRE itea-tvpe correlations were markedly lower than the GRE sub-score 
correlations. This aeant th there was greater unexplained variance on the GRE 
Itea-types than on the GRE sub-scores. Stated another way# less of the student 
achievement aeasured by the SAT was explained the GRE item-type scores than 
explained the GRE sub-scores. GRE Itea-type correlations were frequently 10 
percent weaker than their corresponding GRE Sub-score correlation coefficients. 

Regressing GRE Verbal Item-Types on SAT Verbal sub-scores, GRE Quantitative 
Item-Types on SAT Math scores, and GRE Analytic Item-types on SAT Total scores 
appeared appropriate. In general, the GRE Item-Types and the SAT scores showed 
significant positive correlation, suggesting that the SAT test as comparable 
measure of general learned abilities. GRE Item-type correlations were noticeably 
lower than their corresponding GRE sub-scores, suggesting that i^ndividual 
itea-types also measure discrete abilities apart from those of the SAT 
sub-scores . 

Figure 5. The correlation of GRE itea-types and SAT scores in 3 colleges 



GRE Itea-types 


Code 


SAT 
Verbal 


College #1 

SAT 
Math 


SAT 
Total 


Analogy 


ANA 


.6956 


.3820 


.6482 


Sentence Completion 


SC 


.6121 


.4130 


.6101 


Beading Comprehension 


RD 


.6371 


.4781 


.6599 


Antonyms 


ANT 


.7080 


.2732 


.6001 


Quantitative Comparison 


QC 


.3139 


.7117 


.5712 


Regular Mathematics 


RM 


.4137 


.6553 


.6067 


Data interpretation 


DI 


.2887 


.6321 


.5137 


Analytical Reasoning 


ARE 


.4524 


.5961 


.6012 


Logical Reasoning 


LR 


.5134 


.4622 


.5715 
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GRE Verbal 

GRE Quantitative 

GRE Analytic 


GRE-V 
GRE-Q 
GRE-A 


.8127 
.3490 
.5292 


.4658 
.7657 
.6363 


.7674 
.6227 
.6723 








College #3 




GRE Itea-types 


Code 


SAT 

Verbal 


SAT 
Math 


SAT 
Total 


Analogy 

Sentence Coopletion 
Reading Coaprehension 
Antonyas 


ANA 
SC 
RD 

ANT 


.4257 
.5555 
.4953 
.6164 


.1759 
.3363 
.3288 
.2797 


.3445 
.5189 
.4815 
.5151 


Quantitative Conparison 
Regular Hatheoatics 
Data interpretation 


QC 
RM 
OX 


.3277 
.2585 
.2778 


.7704 
.6941 
.4466 


.6779 
.5908 
.4404 


Analytical Reasoning 
Logical Reasoning 


ARE 
LR 


.3041 
.3818 


.5602 
.5164 


.5285 
.5421 


GRE Verbal 

GRE Quantitative 

GRE Analytic 


GRE-V 
GRE-Q 
GRE-A 


.7176 
.3523 
.6119 


.3922 
.8141 
.6119 


.6431 
.7196 
.5904 








College *4 




GRE Itea-types 


Code 


SAT 
Verbal 


SAT 
Math 


SAT 
Total 


Analogy 

Sentence Completion 
Reading Comprehension 
Antonyas 


ANA 
SC 
RD 

ANT 


.7457 
.6244 
.7359 
.7399 


.2972 
.9700 
.4511 
.3388 


.6365 
.5542 
.7061 
.6551 


Quantitative Comparison 
Regular Mathematics 
Data interpretation 


OC 
RM 
DI 


.3216 
.2470 
.3081 


.6512 
.5649 
.3038 


.5542 
.4603 
.3598 


Analytical Reasoning 
Logical Reasoning 


ARE 
LR 


.5032 
.6324 


.3214 


.6352 
.5769 


GRE Verbal 

GRE Quantitative 


GRE-V 
GRE-Q 


.8714 
.3041 


.4170 
.3805 


.7820 
.3983 
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GRE-A 


.6071 


.5724 


.6955 


MiniauD GRE itea-type 




.2470 


• 

.1759 


.3445 


Maxiiaua GRE ites-type 




.7457 


.9700 


.7061 


Mean GRE itea-type 




.4865 


.4887 


.5615 


MiniBkUB GRE sub-score 




.3041 


.3805 


.3983 


MaxiauB GRE sub-score 




.8714 


.8141 


.7820 


Mean GRE sub-score 




.5728 


.5618 


.6546 



p < .0001 



The internal validity of GRE itea-types can be aaeasured by coaparing the 
inter-correlation coefficients of GRE i tea- types. In the varioue DCP Project 
college saaple groups^ the inter-correlations between 6BE Verbal i tea- types were 
relatively stronger than those between other GRE itea-type scores. Each GRE 
sub-score tended to have higher correlations with the GRE iteo-types 
constructing the sub-score than with GRE itea-types constructing other test 
sub-scores. The analysis ^«f correlations among GRE ites-types shows that the 
itea-types have strong internal validity. 

Figure 6 presents the GRE raw score results for itea-types and sub-scores 
for each of four colleges. College 1 students perforaed very well on the GRE 
exasination. They answered an average 57 of Verbal (GRE-V) questions 
correctly. They answered 49 of 60 Quantitative (GRE-Q) questions correctly^ and 
38 of 50 Analytic (GRE-A) questions correctly. This corresponds to trio following 
GRE converted scores: Verbal- 615, Quantitative - 701, Analytic - 688. One or 
more students attained a perfect score on each itea-types except Antonyas (ANT); 
no student attained a perfect score on aore than one itea-type. 

College 4 students perforaed well on the GRE exaaination. They answered an 
average 47 of 76 Verbal (GRE-V) questions correctly. They answered 33 of 60 
Quantitative (GRE-Q) questions correctly, and 27 of 50 Analytic (GRE-A) 

€ 
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questions correctly. This corresponds to the following GRE converted scores: ' 
Verbal- 522, Quantitative - 472, Analytic - 531. A few students attained perfect 
acorea on the Analogies (ANA) and Antonyas (ANT) ites-types; no students 
attained perfect scores on oore than one itea-type. To deteroine the extent to 
which these students evinced gains over their precollege SAT scores, the GRE raw 
scores were regressed on the corresponding SAT scores. 

Regressing GRE Verbal I tea-Types on SAT Verbal sub-scores, GRE Quantitative 
Itea-Types on SAT Math scores, and GRE Analytic" Itea- types on SAT Total scores 
appeared appropriate. GRE Itea-type correlations were consistently and 
noticeably lower than their corresponding GRE sub-scores in each college sample. 
This suggests that individual itea-types also aay neasure discrete abilities 
apart froa those of the SAT sub-scores and/or it aay reflects their lower 
reliability steaoing fron fewer iteas. 

°^ Itea-Type aean scores aoong college saaples. 

GRE Itea-types Nuaber College 1 College 3 College 4 

of Iteas Mean Mean Mean. 



Analogies 

Sentence Coopletion 
Reading Coaprehension 
Antonyas 


18 
14 
22 
22 


13.11 
11.67 
16.77 
15.17 


8.45 
10.21 
11.50 
10.83 


11.47 
9.81 
13.06 
12.84 


Quantitative Coaparison 
Regular Matheaatics 
Data interpretation 


30 
20 
10 


25.07 
16.33 
7.55 


4.32 
11.13 
5.04 


17.40 
10.84 
4.97 


Analytical Reasoning 
Logical Reasoning 


38 
12 


29.05 
8.71 


20.99 
6.01 


20.52 
6.56 


Miniaun 
MaxiauB 
Mean 


10 
38 
21 


7.55 
29.05 
15.94 


4.32 
20.99 
9.83 


4, . 
7 .52 
.1.94 


p > .0001 


186 


143.43 


88.47 


107.47 
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Calculation of aean score gains for students 

As Figure 6 deaonstrates^ the students in each of four colleges performed 
well on the 6RE General ExaBination^ answering the questions correctly in 
approximately 113 of the 186 GRE itess. GRE raw scores were varied according to 
college and varied widely. Very different distinctions aaong scores appeared 
when the effect of the precollege learning (as measured by the SAT) was removed. 
When the theoretical scores (as predicted by corresponding SAT scores) were 
compared with the students' actual responses (Figure 7), different college 
groups had the greatest amount of unexplained variance (highest residuals) on 
different item-types. Likewise^ different college groups had the lowest amount 
of unexplained variance on divergent item-types. Significant differences in 
unexplained variance existed among college samples. 

The greatest amount of variance in item-type score gains (residuals)^ 
including the greatest standard error and standard deviation, was found in the 
Data Interpretation item-type in the College 3 sample. The lowest amount of 
variance in score gains was evident in Quantitative Comparisons in College 4. 
Variance wfiS partially explained by the number of items included in each 
item-type, but unexplained variance still differed with the college sample. The 
highest unexplained variance for College 1 and College c was in the Logical 
Reasoning item-type, which consists of 12 items. The highest unexplained 
variance for College 3 was in Data Interpretation, which consists of 10 items, 
and the highest unexplained variance in College 4 was in Analogies, which 
consists of 18 items. The lowest unexplained variance was found in Quantitative 
Comparisons for Colleges 1 and 4; this item-type contains 30 items. The lowest 
unexplained variance for College 2 was Regular Mathematics, which contains 20 
items, and the lowest unexplained variance in College 3 was in Analogies, 
encompassing IB items. The improvement in general learned abilities of students 
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at each of the four collegea was explained to a different extent by each of the 
itea-type residuals. Score gains varied by individual college and by itea-type. 
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Figure 7« Regression of SAT scores on GRE iten-type scores 

Dependent Variables: College 1 College ^ College 3 College 4 

GRE Itea-Types Sample *1 Sample «1 Saaple #1 Sample tl 



Sentence Completion 
F Value 
probability>F 

R-Squared (SAT-V explained variance) 
Adj. R*Squared 

Residuals (unexplained variance) 

Analogies 
F Value 
probability>F 

R-Squared (SAT-V explained variance) 
Adj. R-*Squared 

Residuals (unexplained variance) 

Reading Comprehension 
F Value 
probability>F 

R-Squared ( SAT- V explained variance) 
Adj* R*Squared 

Residuals (unexplained variance) 

Antonvtts 
F Value 
probability>F 

R-Squared (SAT-V explained variance) 
Adj* R-Squared 

Residuals (uhexplalned variance) 

Quantitative Coaparison 
F Value 
probability>F 

R-Squared (SAT-H explained variance) 
Adj. R-Squared 

Residuals (unexplained variance) 

Regular Math 
F Value 
probability>F 

R-Squared (SAT-M explained variance) 
Adj. R-Squared 

Residuals (unexplained variance) 

Data Interpretation 
F Value 
probabHity>F 

R-Squared (SAT-M explained variance) 
Adj. R-Sguared 

Residuals (unexplained variance) 



61.724 


111.195 


38.340 


64.258 


.0001 


.0001 


.0001 


.0001 


.3747 


.4572 


.3899 


.3065 


.3686 


.4531 


.3797 


.3037 


.6314 


.5469 


.6203 


.6963 


96.542 


96.878 


75.154 


31.865 


.0001 


.0001 


.0001 


.0001 


.4838 


.4233 


.5561 


.1812 


.4788 


.4189 


.5487 


.1755 


.5212 


.58U 


.4513 


.8245 


70.371 


100.746 


66.818 


46.807 


.0001 


.0001 


.0001 


.0001 


.4059 


.4329 


.5269 


.2453 


.4001 


.4286 


.5190 


.2401 


.5999 


.5714 


.4810 


.7599 


103.530 


98.440 


72.599 


88.236 


.0001 


.0001 


.0001 


.0001 


.5013 


.4272 


.5475 


.3799 


.4964 


.4228 


.5400 


.3756 


.5036 


.5772 


.4600 


.6244 


105.711 


103.606 


44.174 


210.216 


.0001 


.0001 


.0001 


.0001 


.5065 


.4397 


.4240 


.5935 


.5017 


.4355 


.4144 


.5906 


.4983 


.5645 


.5856 


.4094 


77.507 


136.436 


28.126 


133.840 


.0001 


.0001 


.0001 


.0001 


. .4294 


.5083 


.3192 


.4817 


.4238 


.5045 


.3078 


.4781 


.5762 


.4955 


.6922 


.5219 


68.545 


52.473 


6.100 


35.875 


.0001 


.0001 


.0164 


.0001 


.3996 


.2844 


.0923 


.1994 


.3937 


.2790 


.0772 


.1939 


.6063 


.7210 


.9228 


.8061 



Analytic Reaaonino 
F Value 
probabillty>F 

R-Squared (SAT-T explained variance) 
Adj. R-Squared 

S«sl<lu«I« (unexplained variance) 

Logical Reasoning 
F Value 
probability>F 

R-Squared (SAT-T explained variance) 
Adj. R-Squared 

Re«idual« (unexplained variance) 



58.287 
.0001 
.3614 
.3552 
.6448 



49.969 
.0001 
.3267 
.3201 
.6799 



78.858 
.0001 
.3740 
.3692 
.6308 



20.391 
.0001 
.1338 
.1272 
.8728 



40.593 
.0001 
.4035 
.3936 
.6064 



29.926 
.0001 
.3328 
.3217 
.6783 



55.821 
.0001 
.2794 
.2744 
.7256 



59.917 
.0001 
.3328 
.2938 
.7062 



Reqreaaion analvflln of GRE itea-type scores on SAT sub-scorea 

The SAT scores explained saaller portions of variance in GRE itea-type 
scores than in the GRE sub-scores (Verbal, Quantitative and Analytical). The SAT 
Verbal explained fro. 30 to 37 percent of the variance in the Sentence 
Completion itea-type aaong the four college saaples. The SAT Verbal explained 
froa 18 to 55 percent of variation in the Analogies itea-type, froo 24 to 52 
percent of the variation in Reading Coaprehension and froa 38 to 54 percent in 
Antonyaa iteas. The. SAT Math scores explained froa 41 to 59 percent of variation 
in Quantitative Coapariaon itea reaponaea, froa 31 to 50 percent of variation in 
Regular Math itea-type scores, and froa 8 to 39 percent of variation in Data 
Interpretation for the four aaaplea. The regreaaion of Data Interpretation 
scorea on SAT Matheaatics scores waa the only procedure yielding a level of 
significance leas than .0001 (e>.0164). The combined SAT Verbal and SAT Math 
scores (referred to aa SAT Total) explained froa 27 to 39 percent of variance in 
Analytic Reaaonlng and froa 13 to 32 percent of variance in Logical Reasoning 
for the four saaples. In all instances other than Data Interpretation, the 
regression aodel proved significant at the .0001 level, suggesting effective 
control aeasures for the general learned abilities of students as they entered 
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college as freshaen. 

College 1 students entered college with slightly higher aean SAT Math score 
(699) than SAT Verbal score (638). As they approached graduation, these College 
1 students reaained noraatively. stronger in Quantitative abilities (aean GRE-Q = 
701) than in Verbal abilities (»ean GRE-V = 615). Yet, the regression analysis 
showed the College 1 students evinced large gaias on specific GRE itea-types 
that were not accounted for by the comparison of the aeans aggregates 
represented in the sub-scores of the two tests. Specific and significant gains 
weire demonstrated in Analytic Reasoning a Logical Reasoning. These gains 
exceeded those found in the Saaple «1 groups of the other institutions of the 
study. Such findings warrant further investigation of College 1 students general 
learned analytic abilities. 

College 4 students entered college with slightly higher mean SAT Verbal 
scores (543) than SAT Math scores (523). As they approached graduation, these 
College 4 students reaained noraatively stronger in Verbal abilities (aean GRE-V 
= 522) than In Quantitative abilities (mean GRE-Q = 473). Yet, the regression 
analysis showed the College 4 students evinced large gains on specific GRE 
QuantitatiLv, tea-types that were not accounted for by the comparison of the 
aeans aggregates represented in the test sub-scores. Specific and significant 
gains were deaonstrated in Data Interpretation and Quantitative Comparisons. 
These gains exceeded those found in the college saaple groups. Such findings 
- warrant further investigation of College 4 students general learned in abilities 
in Data Interpretation and Quantitative Comparisons. 

Using the student score gains obtained fro. the regression analysis above, 
the mean score gain for each course enrolling 5 or aore students was calculated 
for all the 9 GRE itea-types for each of the four colleges. Such a procedure did 
not assuae that the specific gains of the students enrolled in each course were 
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directly caused by that course. Rather, the acore gains of each student were 
attributed to all the courses in which they enrolled, and the aean score gain 
Cor each course served as a proxy measure of student ..ains. Onca courses are 
clustered by these gains, then hypotheses can be generated and tested as to why 
students who enrolled in a given pattern of courses experienced significant 
gains on one or aora of the outcaraes criteria (i.e., the item-type residuals). 
The procedures of the DCP Cluster Analytic Model, using these aean course scora 
gains (residuals), is outlined in the following section. 



The DCP Project Cluster Analytic Model 
Cluster analytic aodel procedures 

Described below are steps required in the Cluster analytic aodel to assess 
the effects associated with the coursework patterns on the general learned 
abilities of college students. The research design uses as data sources 
transcripts and 6RE and SAT test scores froa a sample of students. The 9 item- 
type categories of the General Tests of the Graduate Record Examination are used 
as measures of general learned abilities of college seniors. These seniors' 
Scholastic Aptitude Test scores are used as variables to control for the aca- 
deaic abilities of these students when they first entered college. The students 
transcripts are used as the record of the sequence of courses in which these 
seniors enrolled. 

The first objective of the cluster analytic model is to deteraine the stu- 
dent gains in general learned abilities over the time of their baccalaureate 
program. To do this, first the residual score of each item-type- for each student 
is calculated; the residual score is the difference between the student's actual 
score and the score predicted by the student's corresponding SAT score. Thus, 
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for each student outcome measure there is a student score gain for each person 
in the sample group. 

The second objective is to determine patterns of coursework on the students 
transcripts which are associated with student score gains. This second objective 
is accomplished using cluster analysis, using their student score gains (GRE 
item-type residuals) as attributes of the courses in which they enrolled. 

A raw data matrix consisting columns of courses and rows of score gains is 
created. The mean score gain for a ll the students in the saaple who enrolled in 
a given eouraa Is calculated and used and hAr-n«Afl the metric value for that 
course. The correlation coefficient is used as the resemblance coefficient ysed 
to transform the data matrix into a resemblance matrix, wherein the similarity 
of score gains for students enrolling in one course can be compared with those 
enrolled in another course. Once the resemblance matrix indicating the propor- 
tional relationship of courses is established, then a clustering method is se- 
lected and executed to arra ge a tree or dendrogram of courses related by the 
student score gains. Next, a discriminant analysis is performed on the resulting 
clusters of coursework to a) determine the extent to which the courses have been 
correctly classified according to the 9 mean student score gains, b) to deter- 
mine which of the 9 mean score gains were correlated with particular discrimin- 
ant functions^ and c) to deterainG which coursework clusters exhibited high aean 
score gains relative to each discriainant function. From the discriainant anal- 
ysis^ then^ can be inferred an association between coursework patterns (clus- 
ters) and general learned abilities (student score gains on 9 criterion vari- 
ables). The cluster-analytic procedure groups courses frequently chosen by stu- 
dents according to the strength of their associated effect on the student score 
gains. 
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Described in greater detail below the steps followed in this cluster ana- 
lytic procedure: 



StgE^. Calculate a student score gain for each itea-type (attribute) of each 
student GRE. This step removes the predictive effect of the student's 
SAT scores froa the GEE iteo-type^ thereby controlling for the .acadeaic 
ability of the student upon entrance to college. For GRE Quantitative 
iteo-types^ the effect of the student's SAT Math score is partialled 
out. For the GRE Verbal itea- types, the effects of the SAT Verbal score 
is partialled out. For the GRE Analytic itea-types, the effect of the 
coabined SAT Verbal and SAT Math scores are partialled out. In this way, 
the student's acadeaic abilities prior to entering college is controlled 
when calculating student score gains. 

StepJ,, Calculate the aean score gain for each course enrolling 5 or aore stu- 
dents froa the saaple group. Cross-listed courses (courses with aore 
than one possible course nusber) are standardized so that they have only 
one identifier. Courses with the saae course identifier but with absten- 
tiously different content (i.e., "Music 101: Voice" and "Music 101: Pi- 
ano") are excluded from the analysis. Likewise catalog changes are ac- 
counted for. If Math 201 in 1982 was renumbered as Math 211 in 1985, 
Math 201 and Math 211 for those years are treated as the saae course for 
the purposes of analysis. 

The proportion of courses included in the analysis is related to a) the 
extensiveness of the course listings in the curriculua and b) the size 
of the student sample. The aore extensive the curriculua, the less fre- 
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quently 5 or aore students froa a sample groups will have enrolled in 
the sazse course. Likewise* the soaller the size of the student sample* 
% the less frequently 5 or aore students froa a saaple groups will have 

enrolled in the saae course* 

Step 3 . Create a raw data matrix. The data matrix consists of columns repre- 
senting of all the courses (objects) appearing on 5 or aore student 
transcripts in the sample. The rows in the data matrix consist of the 
student gain (6RE item-type residual) scores. The aean residual score 
associated with that course and that student outcome measure is entered 
into each cell. For example* the course (object) in the first column in 
the data matrix is AHTKB0P0L06Y 101* and student outcome measure in the 
first row of the data matrix is DATA IHTERPRETATIOH. The student score 
gains are .40* .45* .50* .55* and .60; the aean score gain* therefore* 
is .50 and is entered as the aetric variable in cell (1*1) of the ma- 
trix. Since the variables in each row are of the saae aagnitude* and 
therefore* have comparable effect on the resulting cluster analysis^ the 
data matrix does not need to be standardized (Romesburg* 1984). The 
cluster analysis will taxonomize courses in the curriculum according to 
whether students who showed positive gains on each itea-type were enrol- 
led in the courses. This step prepares a raw data aatrix to be used in a 
general cluster analysis based on quantitative data. 

Step 4 . Select a resemblance coefficient. The resemblance coefficient 

(Romesburg^ 1984) is also called the similarity index (Lorr* 1983). The 
purpose of the resemblance coefficient is to explain the similarity (or 
dissimilarity) of each cell to each of the other cells in the data mat- 
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rix; it is expressed aathe«atically. There are aany resemblance coef- 
ficients; each will express the siailarity between courses (objects) in 
a slightly differently way. Each coefficient is appropriate for achiev- 
ing slightly different research goals. 

The resemblance coefficient selected is Pearson's product-monsent cor- 
relation coefficient. It is appropriate for use with ratio data. For 
each course (object), it vill calculate its siailarity to each other 
course according to the 9 itea-type residuals (attributes) coded in the 
data aatrix. The reseablance coefficient expresses the relationship of 
two courses proportionally. 

SteE_S. Calculate « resemblance aatrix froa the raw data oatrix. The reseablance 
aatrix is calculated by transforming the raw data matrix using the cor- 
relatior resemblance coefficient. In this cluster analytic model, the 
data matrix consists of quantitative data described by 9 attributes 
ranging in value froa 1.00 to -1.00. In the reseablance aatrix, the col- 
umns represent the first course (object) in a pair, the rows represent 
the second course (object) in a pair. The resemblance coefficient (Pear- 
son's r) is entered into each cell. The cell value represents the extent 
to which the attributes on the first course explain the variance in at- 
tributes on the second course. The reseablance coefficient serves a mea- 
sure of similarity between one course and each other course in the cal- 
culation of clusters or coursework patterns. 

Stee_6. Select and execute the clustering method. A resemblance matrix is trans- 
formed into a tree of related courses (objects) by use of a clustering 
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aethod* A clustering method is a series of steps that removes values 
froa the resemblance matrix, thereby reducing the size of the matrix. 
Each time a value is removed from the resemblance matrix it is placed in 
the cluster tree or dendrogras. In the last step, the resemblance matrix 
disappears completely and the tree is completed as the last value in- 
serted* 

Romesburg (1984, p* 139) recommends using the unweighted pair--group 
method using arithmetic averages, also known as the average linkage 
method, which is abbreviated as UP6MA. UP6MA is recommended over single 
linkage clustering method (SLINK) and complete linkage clustering method 
(CLINIC) for two reasons. First, it can be used with any resemblance co- 
efficient, while SLINK and CLINK are designed to be used with interval 
and ratio data in a quantitative data matrix. Second, it judges the sim- 
ilarity between pairs of clysters in a less extreme manner than do SLINK 
and CLINK. The average linkage method (UPGMA) is available on SPSSx, SAS 
and BMDP statistical packages. 

Step 7 . Determine the optimum number of coursework clusters. Cluster analysis 

is a procedure for taxonoaizing or classifying coursework data. The num- 
ber of groups or patterns in which the data is classified according to 
the criterion variables is an arbitrary one. Once relationships between 
courses have been determined, the researcher must decide on how many 
groups in which to put the data. Discriminant analysis provides a means 
to test the secondary validity of the coursework pattern groupings. 
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By coBputing successive cluster analyses for different numbers of clus- 
ters and then conducting discriainant analyses on the resultant group- 
ings, one can identify the nuaber of clusters with has. the highest pre- 
dictive value, given the criterion variables used. Using the DISCRIMIH- 
AHT prograa in SPSSx, for exaople, will identify how aany members of 
each coursework pattern or cluster were correctly classified, how many 
could be classified in other patterns, and what was the overall percen- 
tage of correct classification. 

The number of clusters with the highest predictive value may not be the 
sole objective in examining the merits of different cluster solutions to 
the cluster analysis. Theoretically, a 4-cluster solution may have high 
predictive value for GRE item-types because the item-type residuals are 
forced into three discriminant functions which should approximate the 
GRE sub-scores. Likewise, a 13-cluster solution may prove to be a 
slightly less predictive, but the 9 GRE item-type residuals may be more 
clearly associated with discrete coursework patterns. Careful visual 
inspection of the cluster dendrogram often suggests appropriate cluster 
solutions to test using discriminant analysis. 



Ste£^. Determine which criterion variables contribute significantly to which 
discriminant functions. DISCRIMIMANT in SPSSx, for example, calculates 
the pooled within-groups correlations between the discriminating vari- 
ables (in this case, the mean residual scores on the 9 item- types) and 
the canonical discriminate function. Large positive and negative cor- 
relations are identified. Next, the group means for each coursework 
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cluster can be examined. In this manner, the patterns of coursework as- 
sociated with one or more isean item-type residual scores can be identi- 
fied. 

SteE_9. Repeat Steps 1 to 8 using a second cohort of students • Following Steps 1 
through 8 will produced a set of hypothesized relationships between 
coursework patterns and student score gains on 9 criterion measures of 
general learned abilities. Hypothesized relationships cannot be tested 
or validated using the same data. Therefore^ a second institutional sam- 
ple is drawn. A second group of students are tested and a second set of 
transcripts and student score gains are evaluated. Repeated use of the 
model should refine and clarify members within each coursework pattern • 

Through the above 9 steps, the cluster analytic model classifies the most 
frequently enrolled courses according to their associated effect on student 
score gains. The Cluster Analytic Model classifies courses according to a ratio 
index of similarity to other courses. While the Cluster Analytic Model may 
examine only a fraction of all the courses in a college curriculum, it does 
provide a means to differentiate the effect of required courses or courses in 
which most student enroll. For example, a sample of 105 College 1 transcripts 
yielded 303 individual courses enrolling 5 or more of the students from a list 
of 1,445 unduplicated courses. While these 303 courses represented about 
one-fifth of all courses on the transcripts, a majority of the courses were 
those listed in the College 1 catalog as meeting the general education 
requirements of the University. 
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Creating the raw data matri v and the resemblance matr-iv 

For each college saople, the mean score gains for each course were 
calculated. In College 3, for example, 149 courses were found on 5 or more of 
the student transcripts. A raw data matrix was created consisting of 149 columns 
and 9 rows (149 x 9). The rows represent the criterion variables: the mean 
student score gains on the 9 GRE item-type scores, while the columns represent 
those courses enrolling 5 or more students. Thus, each cell value of the matrix 
is a mean GRE item-type score gain for those sample group students enrolling in 
a specific course. The procedure was employed for each college sample. 

A resemblance matrix was created next to describe how closely each course 
resembles the other courses according to the criterion variables: the student 
score gains. To calculate the resemblance matrix, the correlation coefficient 
was selected as a similarity measure. The correlation coefficient is the Pearson 
product-moment correlation coefficient. Thus, this coefficient assesses a pat- 
tern similarity of any two courses explained in teres of the 9 GRE item-type' 
score gains. 

For College 3, the resemblance matrix produced in this .step consisted of 
149 rows and 149 columns (149 x 149), in which each cell value theoretically 
ranges from -1.00 to 1.00. The calculation of the resemblance matrix was done 
using the SPSSx PROXIMITY prograa. The program provides 37 different proximity 
measures. Ten are for quantitative data and the remainder are for binary or 
qualitative data. This program can directly produce distance, dissimilarity, or 
similarity matrices as text files for a small to moderate numb<^ ■ of cases and 
variables, which can be directly used for other SPSSx procedures or for other 
statistical programs. BMDP PIM can also be used to calculate the correlation 
resemblance matrix and save it as a text file. 
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Selection of the clustering aethod 

The method selected for this quantitative analysis was the average linkage 
aethod (UPGMA). A sumoary of the results of the cluster analysis of the College 
3 sample are presented in Figure 7. Courses were classified into 13 coursework 
patterns according to a hierarchical cluster structure. In fact, the choice to 
present the data in 13 clusters is arbitrary. Any number of clusters can be 
i-lentified depending on the hierarchical cluster structure produced; this 
structure reoains constant regardless of the number of clusters used to fora 
coursework patterns. A procedure for selecting the optimum number of clusters 
and for validating the resulting patterns will be described in greater detail in 
a subsequent section on the discriminant analysis of the coursework patterns. 

Using a 13 cluster solution to the quantitative cluster analysis, the 
largest number of courses are found in Coursework Cluster #1 with 44 courses. 
The smallest clusters are the 13th cluster with 1 course and the 7th, 10th and 
11th clusters with 2 courses. Overall, the differentiation between clusters is 
attributable to the number of criterion variables used in the analysis and also 
to the choice of those variables. The cluster analysis and subsequent 
discriminant analysis suggested that student score gains on GRE item-types are 
strong, reliable and robust measures in differentiating student general learned 
abilities. 

In the cluster analysis developed for each of the four college samples, a 
careful examination of courses within each cluster indicated that some courses 
from tha same department tend to be combined together. Also, a set of courses 
coming from various related disciplines may form a homogeneous cluster on the 
basis of a set of given attributes or criteria. At this point in the analysis, 
it was difficult to describe which dimensions of student general learned ability 
each clusters represents. However, it seemed to be clear that one pattern of 



47 - 



course enrollnent aay contribute to student general learned ability in a way 
significantly different from the other coursework patterns and that those 
patterns varied from college to college. Supporting this is a more detailed 
examination of subset courses of each clusters. In many cases, those courses 
offered at the same level tend to be combined into pairs together. But, those 
pairs are agglomerated with other courses offered at the higher level again 
according to the hierarchical structure of clusters. This may suggest that 
student gains in general learned abilities may be obtained through a sequential 
enrollment pattern during the college years, not at a single stage of the 
sequence (such as the freshman year experience). 

A sample hierarchical cluster structure is presented for College 3 in the 
dendrogram of Figure 7b. For concise visual presentation, the complex 
sub-structures of each of the clusters w«^re omitted from the dendrogram in 
Figure 7b. The dendrogram displays the clusters being combined and the distances 
between the clusters at each successive steps, suggesting that the 13-cluster 
solution examined is appropriate and interpretable. Cluster analyses using 
smaller and la. .^er numbers of cluster groupings provided comparably high levels 
of correct classification, as determined by subsequent discriminant analyses. 
However, as the resemblance index increases (Euclidean distance between 
courses), Qore -distant courae^i are joined ii to larger and larger clusters. A 
5-cluster 3o?.v;tion, ^or example, provides^ s high degree of aggregation which may 
prove to have a high degree of predictive validity but a lot tvel of utility in 
differentiating coursework by item-type. 
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Figure 7-c. Dendrogram of hierarchical cluster structure for College 3. 

«K==a=s:=s=s=s=sssss=3--— „^ ^ 



ECOH -49( 45 

CCON -136 40 

ECOW -7;^ 48 

£CON *-190 44 



'HT"?*^ i^/^ i -l_ f ■ - h: J • •"r'**""T**— •••• ••••.-41 




fiZO 

ARTS -9l| 
HMS -lOtt 



ERLC 



- 49b 



53 



Figure 7-c. Dendrogram of Hierarchical cluster structure for College 
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following a course indicates a course misclassified according to the dis- 
criminant analysis of course clusters. 
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Figure 8b. Courses within coursework clusters (13-cluster solution): College 3 
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"** following a course indicates a course misclassified according to the dis- 
criminant analysis of course clusters* 

A careful exanination of courses within each cluster seeas to indicate that 
some courses cooing froa the saoe departsent oay appear in the sane cluster^ 
such as the Econoaics courses (Econ) in Cluster #1* Sisilarly^ there are 
apparent sequences of courses, such as the Cheoistry 101, 102, 103, 104 sequence 
in Cluster #!• Also, a set of courses coming from various related disciplines 
may form a homogeneous cluster on the basis of a set of given attributes or cri- 
teria. There appears to be a homogeneity of humanities and social science 
disciplines in Cluster #2. 

At this point in the analysis, it is difficult to describe which dimensions 
of student general learned ability are represented in each cluster. However, it 
seems to be clear that one pattern of course enrollment may contribute to stu- 
dent general learned ability in a way significantly different from the other 
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coursework patterns. Supporting this is a oore detailed exaaination of subset 
courses of each clusters. In aany cases, those courses offered at the saoe level 
tend to be coabined into pairs together. But, those pairs are aggloaerated wi.th 
other courses offered at the higher level again according to the hierarchical 
structure of clusters. This aay suggest that student gains in general learned 
abilities Bay fas obtained through a sequential enrollment pattern during the 
college years, not at a single stage of the sequence (such as the freshaan year 
experience) . These observations were repeatedly conveyed in the analysis of each 
of the four college saaples. 

Diacria inant analysis of coursework patterns 

In exaaining the dendrograa of coursework, a logical question arises as to 
which nuaber of clusters or pattern groupings provides the best explanation of 
the relationship between student itea-type gain scores and coursework patterns. 
Separate discriainant analyses of different nuobers of cluster groupings were 
performed for each college saaple in order to deteraine the number of groupings 
that optimizes the proportion of courses correctly classified. Four different 
cluster solutions provided comparably high levels of correct classification (See 
Figure 9): 



Figure 9. Percent of correct classification by college and cluster solut 



ion. 



Percent of courses correctly classified 
according to Discriminant Analysis of Cluster solution 

CoHege 1 College 2 College 3 College 4 

5 cluster solution : 89.92% 89.93X 87.65X 

8 cluster solution : 82.48X 90.07X 92.99X 87 41% 

13 cluster solution : 85.48X 89.36X 94.63X 85.43X 

25 cluster solution : 83.83X 86.17X 90.45X 83 70X 
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Accuracy veraus explanation 

While these cluster solutions produced comparable classification results^ 
the different grouping evidenced differing effectiveness in identifying rela- 
tionships between mean iteo-type residuals (gain scores) and coursework pat- 
terns. The 13-cluster solution proved to provide the greatest extent of informa- 
tion about such relationships and was therefore used in this report. 

The discriminant analysis was conducted using the DISCRIMINANT program in 
SPSSx in the following manner. Using the course item-type attributes as inde- 
pendent variables and the cluster group membership as the dependent variables^ 
discriminant functions were applied to the data. The discriminant functions and 
the courses item-type mean residual scores were used to see how correctly the 
discriminant function identifies each cluster group. The resulting percentage of 
correct predictions serves as a secondary validation of the cluster solution 
(Bradfield and Orloci^ 1975; Green and Vascotto, 1978; Romesburg^ 1984). As Fig- 
ure 9 demonstrates^ the DCP Cluster Analytic Model consistently grouped courses 
correctly according to the 9 criterion variables. Figure 10 provides an example 
of the classification breakdown for College 3 as an example. 
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Figure 10. Discriainant analysis of the 13-cIuster solution for College 3. 
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Percent of 'Grouped' Clusters correctly classified: 94.632 
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Correl ations of item-tvoes and discriminant functions 

The discriminant analysis of College 3 Sample #1 provided secondary valida- 
tion that 94.63X of the classification of courses was correctly predicted by the 
cluster analysis (Figure 10). The discriminant analysis is a secondary valida- 
tion, since it is based on the same sample of transcripts and test scores; pri- 
mary validation of the cluster analysis may be achieved through the replication 
of coursework patterns in the College's Sample #2 to be analyzed in the fall of 
1989. 

Stated simply, over 9 of 10 courses more frequently taken by College 3 Sam- 
ple #1 students were correctly classified according to their mean residual 6RE 
scores. Bhile the cluster analysis produces coursework patterns according to 
criteria of general student learning, additional steps are needed (1) to deter- 
mine which courses were correctly classified and (2) to ascertain which item- 
type scores contributed to any given coursework pattern. 

Using the BREAKDOWN procedure in the DISCRIMINANT program of SPSS-X (Noru- 
sis, 1985)/ courses which were incorrectly classified or which may be classified 
within another coursework pattern are identified. When a set of data is taxono- 
mized by multiple criteria, it is reasonable to expect that the first groupings 
are generally the most homogeneous and the later groupings are the most hetero- 
geneous. Consequently, for College 3, the highest proportions of misclassifica- 
tions occurred in Cluster *10, while all other Clusters had relatively few mis- 
classifications (See Figur(i3 8a-8b, 9). 

To compute the contribution of each mean item-type residual score to the 
discriminant functions, the correlation coefficients between mean residual 
scores and discriminant functions were examined. 
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Correlations o£ coursework clusters and discriminant functions 



Exaaination of the relationship between 6RE itea-type residuals and dis-* 
criainant functions revealed the following pooled within-group correlations over 
•50: 

Function 1 was positively correlated to Quantitative Cosparisons (r=,61); 
Function 2 was positively correlated to Analytic Reasoning (r=,77); 
Function 3 was positively correlated to Antonyas (r=,64); 
Function 4 evinced no strong correlations; 

Function 5 was positively correlated to Data Interpretation (r=,S2), but 
was negatively correlated to Sentence Coapletion (r--*64); 

Function 6 evinced no strong correlations; 

Function 7 was positively correlated to Analogies (r=.S2); 

Function 8 was positively correlated to Regular Mathematics (r=.72); 

Function 9 was positively correlated to Quantitative Cooparisons (r=,54)# 
was positively correlated to Reading Coaprehension (r=.63), and 
was positively correlated to Logical Reasoning (r=.75). 

Once the relationships between discrizainant functions and aean itea-type resi-* 

duals have been established, then the relationships between the discriainant 

functions and the coursework clusters can also be deterained. 

By exaaining the average score of each cluster group for each' discriainant 

function, the extent to which each discriainant function contributes to that 

group Bay be calculated. The average residual score for a coursework cluster 

group is called the group centroid. Group centroids for each coursework cluster 

in the College 3 are presented in Figure 11 as an exaaple. 
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Figure 11. Canonical diacrieinant tjjnctionfl evaluated at group leans: College 3. 
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Interpreting the coursework clusters for the 13-clu3ter solution 

Figure 11 shows the coursework cluster aeans (group centroids) for each 
discriminant function for College 3. Clusters with positive or negative aeans 
greater than 1.0 were selected for further analysis. 

Coursework Cluster tl had a positive group sean on Functior 2 and a high 
negative group aean on Function 1. Function 1 was positively correlated to Quan- 
titative Comparisons (r=.61)^ and Function 2 was positively correlated to Ana- 
lytic Reasoning (r=. 77) .Students enrolling in the coursework of Cluster #1 
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showed iaprovement in Quantitative Comparisons and decline in Analytic Reasoning 
abilities* 

In contrast/ Cluster #2 had a high positive group mean on Function 1. This 
Cluster also evinced high negative group means on Function 3 and Function 4. 
Function 3 was positively correlated to Antonyms (r=.64), while Functioi, 4 ex- 
hibited no strong correlations* Students enrolling in Cluster #2 coursework 
showed improvement in their Quantitative Comparisons abilities ^ but dipped in 
their ability to answer Antonyms questions correctly. 

Cluster #3 had a high positive group mean on Function 1 and a high negative 
group mean on Function 7. Function 1 was positively correlated to Quantitative 
Comparisons (r=.61). Function 7 was positively correlated to Analogies (rs.52). 
Students enrolled in this coursework cluster showed improvement in Quantitative 
Comparisons but declined in Analogies. 

Cluster S4 had a high positive group means on Functions 1, 6, and 7. Clus- 
ter #4 also evidenced high negative group means on Functions 2 and 3. Function 1 
was positively correlated to Quantitative Comparisons (r=*61). Function 2 was 
positively correlated to Analytic Reasoning (r==.77), while Function 3 was po^ii- 
tively correlated to Antonyms (r=.64). Function 6 evinced no strong correla- 
tions. Function 7 was positively correlated to Analogies (r==.52). Therefore, 
students enrolling in this coursework bettered their performance in Quantitative 
Comparisons and Analogies, but flagged in Antonyms and Analytic Reasoning. 

Cluster #5 had a high negative gyoup mean on Functions 1 and 3. Accord- 
ingly, students who enrolled in the coursework pattern represented in Cluster #5 
displayed declines in their ability to make Quantitative Comparisons and to de- 
cipher Analogies. 

Cluster *6 had a high positive group mean on Function 8 and high negative 
group means on Function 3 and 4. Function 3 was positively correlated to 
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Antonysa (r=.64), while Function 4 exhibited no strong correlations • Function 8 
was positively correlated to Regular Matheoatics (r=.72). Students enrolled in 
Cluster 16 coursework showed gains in Regular Mathematics but evinced decline in 
their ability to identify Antonyms* 

Cluster *7 had a high positive group mean on Function 9 and high negative 
group means on Functions 3 and 5. Function 3 was posittively correlated to Ant- 
onyms (r=.64). Function 5 was positively correlated to Data Interpretation 
(r=»52), but was negatively correlated to Sentence Completion ir=-,64). Function 
9 was positively correlated to Quantitative Comparisons (r=,54), Reading Compre- 
hension (r=»63), and Logical Reasoning (r=.75). Consequently, students regis- 
tered in Cluster #7 coursework demonstrated gains in Reading Comprehension, Sen- 
tence Completion, Quantitative Comparisons and Lo^iical Reasoning. They also 
showed declines in abilities in Antonyms and Data Interpretation. It should be 
noted that Cluster #7 had but 2 members, albeit that the discriminant analysis 
indicated that both were correctly classified (See Figure 9). 

Cluster #8 produced a positive high group mean on Function 1 and high nega- 
tive group means on Functions 2 and 5. Function I ms positively correlated to 
Quantitative Comparisons (r=.61), while Function 2 was positively correlated to 
Analytic Reasoning {r=»77). Function 5 was positively correlated to Data Inter- 
pretation (r=»52), but was negatively correlated to Sentence Completion 
(r=-»64). Students listed on the rolls of Cluster #8 coursework evinced im- 
provement in Data Interpretation Quantitative Comparisons and Sentence Comple- 
tion. They exhibited declines in their ability to answer Data Interpretation and 
Analytic Reasoning questions. 

Cluster #9 showed a high positive group mean on Function 3. Function 3 was 
positively correlated to Antonyms (r=.64). Students in this coursework improved 
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in their abllltlos on this 6RE Itea-type. 

Cluster #10 had only 2 courses as members^ one of which was misclassified 
(See Figure 9). Under such circumstances further analysis was not appropriate* 

Cluster #11 offered a high positive group aeans on Function 2^ 3^ 5^ 7 and 
9 and a high negative group oean on Function 4» While Cluster §11 had only 2 
cembers^ both were correctly classified (See Figure 9)» Function 2 was posi- 
tively correlated to Analytic Reasoning (r=.77)^ while Function 3 was positively 
correlated to Antonyas (r=»64)» Function 4 exhibited no strong correlations. 
Function 5 was positively correlated to Data Interpretation (r=»52), but was 
negatively correlated to Sentence Coapletion (r=-.64)» Function 7 was positively 
correlated to Analogies (r=»52)» Function 9 was positively correlated to Quanti- 
tative ConparAsons (r=.54)^ Reading Coaprehension (r=»63)^ and Logical Reasoning 
(r=*75). Students enrolling in Cluster #11 coursciwork improved in their ability 
to correctly answer Antonyms^ Analogies, Reading Coaprehension, Ouantitritive 
.Comparisons^ Data Interpretation, Analytic Reasoning, and Logical Reasoning 
questions, but dipped in their Sentence Completion abilitieti. 

Cluster #12 had a high positive group aean on Function 2. Function 2 was 
positively correlated to Analytic Reasoning (r=.77). Students enrolled in this 
Cluster improved in their * ility in correctly answering Analytic Reasoning 
questions. 

Cluster #13 had but one member, albeit that member was correctly classi- 
fied. Cluster 13 had a high negative group mean on Function 9 and a -high posi- 
tive group oean on Function 5. Function 5 was positively correlated to Data'^In- 
terpretation (r=.52), but was negatively correlated to Sentence Completion 
(r=-.64). Function 7 was positively correlated to Analogies (r=.52). Function 9 
was positively correlated to Quantitative Comparisons (r=,54), Reading Compre- 
hension (r=,63), and Logical Reasoning (r=.75). Students enrolled in this 
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cour^ework cluster declined in Reading Cooprehension^ Quantitative Comparisons 
and Logical Reasoning and iaproved in Data Interpretation* 

Figures 12a - 12f portray the College 3 coursework clusters arranged ac- 
cording to the GRE itea-type with which they were associated. It should be cau- 
tioned that the association was established at the cluster level. No direct 
causal link was intiaated between student enrollment in any one given course and 
scores on the GRE. Furtheraore^ at this pointy one cannot say why s\.udents who 
enrolled in these courses had higher score gains. The cluster serves to hypothe- 
size relationships between coursework patterns and the general learned abilities 
measures by the item-types of the GSE. One can say that students who enrolled 
in specific patterns of coursework tended to perform better on specific item- 
types within the GRE, while others who enrolled in different coursework patterns 
did not tend to perform as well. Comparable analyses of the other college sam- 
ples produced similar relationships. 
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Figure 12a. Coursework clusters associated with high positive laean residuals. 
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discriminant analysis of course clusters. 
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clusters asseciated with Mgh positive mean residuals. 
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Figure 12c. Coursework clusters associated with high positive Bsean residuals. 
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criminant analysis of course clusters. 
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Figure 12d» Coursework clusters associated with high negative taean residuals. 
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Figure 12e. Coursework clusters associated with high negative mean residuals. 
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Figura 12f* Coursework clusters associated with high negative aean residuals. 
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criminant analysis of course clusters. 
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Observations about the cluster relatlonshipa 

The coursework patterns resulting from the cluster analysis do not produce 
clear distinctions according to academic departoent or aajor. This was true in 
all saoples exaained. Departments and programs, establish different courses to 
accomplish varying purposes. It is not surprising to find a "history of 
course in aost fields of s^udy. Likewise/ it is not unusual to find a "research 
Methods* course in most disciplines. It should be remembered as well that many 
of the courses established by a department/ program or division are designed 
primarily to impact specialized knowledge/ skills and abilities rather than 
those associated with general learning. So the coursework associated with stu- 
dent score gains in a given type of general learning does cross disciplinary 
lines . 

In College 3 courses numbered 0 to 99 are lower division courses; courses 
numbered 100 and above are intended for junior and senior students. Several of 
the clusters have equal proportions of lower division and upper division course- 
work. Those clusters associated ^ith Analytic Reasoning draw heavily from upper 
division coursework in Chemistry (CHEM)-/ Economics (ECON)/ Math and Computer 
Science (H&CS). The convention of limiting the stated general education curricu- 
lum to lower division coursework may not be warranted. 

Differt5i;t colleges emphasize or provide a different balance to the general 
learning that occurs. It is clear in College 3 that gains in Analytic Reasoning 
and Quantitative Comparisons predominate. While the regression analysis evinced 
large gains in Data Interpretation in College 3', the relationship between these 
gains and specific coursework patterns was not as apparent • Similar observations 
were found in the cluster analysis of other college samples. While large amounts 
of unexplained variance on one or more 6RE item-types may appear through regres- 
sion techniques/ the cluster analytic procedure with subsequent secondary vali- 
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dation through discriminant analysis provides a aeans of associating the formal 
curriculum with gains in specific learning. Where such relationships are not 
apparent, student score gains also may be attributable factors not addressed in 
the analysis: the reliability of the aeasures themselves, the background of the 
students enrolling in the courses, the extracurricular environmental variables. 

In our analysis of the clusters of coursework in the four colleges and uni- 
versities exanined here, the following conclusions proved to be consistently 



true: 
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1. Developaent of general learned abilities does not have an 
exact one-to-one relationship with departmental catego- 
ries. All quantitative reasoning developaent does not 
occur exclusively in Matheaatics classes. Consequently, 
siople counts of the number of credits or courses a stu- 
dent has taken in a particular subject aay not be a reli- 
able proxy of general learning in the attendant subject 
area. Quantitative skills^ for exaaple^ sany be developed 
in a variety of subject areas. 

2. Developaent of general learned abilities is not confined 
to the lower division. This finding was consistent 
throughout the four colleges exaained. General education 
requirements of colleges should be re-exasined in light 
of student gains in. general learned abilities. Coursework 
that students who showed significant gains took should be 
exaained^ evaluated and incorporated into the general 
education sequence of the college. 

3. There is little foraal aonitoring and description of the 
curriculua in teras of general learned abilities at the 
college-wide or university-wide level. Colleges should 
regularly aonitor the nuaber of credits and courses in 
their curriculum. Without this baseline data^ the extent 
to which student share a coaaon learning experience at a 
college cannot be readily determined. 



The relationships established through the cluster analytic model are aaso- 
ciational, not causal. Once a set of courses has been linked to score gains in a 
specific learned abilities, a targeted investigation can be launched to deter- 
sine the commonalities of teaching-learning environment, of student and faculty 
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expectations of performance^ of the specific abilities of the students who en- 
roiled in the classes. But regardless of uhat hypotheses are generated about why 
this coursework is associated with gains in learned abilities^ one can state 
with confidence that students who enrolled in this coursework demonstrated gains 
on a specific type of learned ability. 

Conclusions 

We began this narrative with the examination of transcripts and assessment 
test scores froa samples of graduating seniors at four colleges and universities 
in the 1987-88 academic year. From this data^ we explored the merits of using 
incoming student test scores (in this case^ SAT scores) as a means for monitor- 
ing students lecrning prior to college. We compared the conclusions about under- 
graduate general learning that might logically be drawn from the examination of 
simple GRE mean scores with those that might result from analysis of GRE residu- 
als^ once the effect of pre-college learning had been removed. In rarforming 
this comparison^ we also noted the apparently greater diagnostic and explanatory 
ability of GRE itea-type scores over the aggregate GRE sub-scores which are con- 
ventionally reported to colleges. Given the way that many standardized pre- 
college and post-baccalaureate tests are constructed^ there appeared to be con- 
sequential value in the use of item-types in existing tests as part of a 
constellation of measures of general learned ability. 

We find redeeming value to these standardized tests for a variety of rea- 
sons: 

1. They provide accessible precollege and postcollege measures without 
superimposing additional testing on students; 

2. They offer established measures with known relationships which clearly 
measure both collegiate and precollegiate learning; 

« 
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3. The itea-tyfes of these tests show some proaise for use as diagnostic 
indicators of specific learned abilities of students; 

4. The known "higher order reasoning" and "critical thinking" measures 
evince strong correlations with the SAT, the GRE or both, calling into 
question the efficacy of alternate instruments in measuring general 
learned abilities. 

What is needed to explore the use of standardized measures, such as the GRE, SAT 
and ACT for assessment purposes is to first device a rationale and mechanism for 
the testing services to provide colleges item-type data on an optional basis. 
Second, a longitudinal 3tudy of 3 to .5 years is needed to determine which among 
the iteE-types has the statistical independence, reliability and construct va- 
lidity to be incorporated as p^irt of an assessment plan. 

In this study we looked not only at test scores but also transcripts. We 
found them to be useful, usable, non-obtrusive measures of what students chose 
to constitute their undergraduate education. We drew samples from 10 to 48 per- 
cent of the graduating seniors at four separate colleges and universities. It 
appeared froa our analysis that a sample size of at least 10 percent was re- 
quired in order to have more of the generf^l education coursi;work adequately rep- 
resented on the student transcripts. Furthermore, we found th^t the extent to 
which students at a particular college held a common intellectual experience (by 
enrolling in similar coursework patterns) was not a function of institutional 
size. The percent of unduplicated coursework taken by 5 or more students in any 
one of the four colleges ranged from 15 to 36 percent. The balance of these stu- 
dents coursework was unique to their program of study, major, minor and elec- 
tives* At these institutions, the general education requirements comprised froa 
one-third to one-half of the coursework required for graduation. This data sug- 
gested that it makes little sense to attempt to make generalizations about the 
quality of the undergraduate educational experience at these institutions; the 
transcript analysis suggested that students have far too diverse an educational 
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experience to make meaningful statementi* about the undergraduate curriculun as a 
whole* 

In contrasts the DCP cluster analytic model provides a consistently reli- 
able means for associating segments of the college curriculum which are associ- 
ated with a particular group of students, with a particular type of learning 
(i»e.# criterion measure) or both* While our student samples were purposely 
drawn at random, students could be stratified by incoming ability, thereby pro- 
ducing coursework patterns associated with gains among high or low ability stu- 
dents* Similarly, the criterion measures could include measures of critical 
thinking, writing analyses r assessments of goal clarity or learning with the 
discipl'nes. These measures need not be chosen to the exclusion of the tradi- 
tional standardized measures of cognitive development, but could be added to 
then to form the constellation of measures which seems to most accurately de- 
scribe the educational experience of the students at a given college or set of 
colleges* To incorporate a variety of different measures into the Cluster Ana- 
lytic Model, one would only need to stati-gtically standardize student scores 
prior to performing the cluster analysis in order to insurs that no one measure 
would predominate by virtue of the scale of evaluation it used. All measures 
would need to have a pre-college assessment counterpart, so that learning prior 
to college could be controlled and so that the assessment endeavor measures 
learning rather than admissions selectivity. 
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